Visual Analysis of Multimedia Data

One of the most important applications in visual analytics has been exploratory visual analysis of large collections of unstructured text documents. However, digital media, especially those on the Internet, are multimedia in content with text, images, video, and even sound together. Furthermore, there is an explosion of broadcast and other media, especially in third world countries. (In the Middle East, for example, there has been a rapid increase in the number of broadcast channels in the last five years.) Thus there is the need to extend exploratory visual analysis to large collections of multimedia. However, this requires intelligent automated analysis techniques closely integrated with interactive visualization. In the case of broadcast video, correlated image segmentation, feature identification, image flow analysis, audio analysis, and closed caption concept extraction are required. In this talk, I will discuss recent work that focuses on analyzing video news broadcasts from several channels over extended periods of time. The news segments are found automatically, and broadcasts in any language can be analyzed. The resulting multiple terabyte database can then be explored visually over time, permitting the user to follow themes as they grow or recede, compare different viewpoints on the same theme, and do many other analyses. I will discuss how this approach can be extended to other types of multimedia.